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Abstract 



We examine the experimental observability of the decay mode t ^ eg at 
the Fermilab Tevatron via the flavor-changing neutral current vertex teg. We 
find that with the existing data, one should be able to probe the teg coupling 
to a value smaller than indirect limits previously obtained from b ^ s^ and 
the measured branching fraction for t — > bW, reaching BF{t — > eg) ~ 15% - 
A data sample of 1 fb^^ (10 fb^^) at y/s = 2 TeV may probe the t ^ eg 



0. 



branching fraction to a level of 2% (0.5%). 
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I. INTRODUCTION 



With the discovery of the top quark H the long anticipated completion of the fermion 
sector of the standard model has been achieved. Its unexpected large mass in comparison 
with the other known fermions suggests that the top quark will play a unique role in prob- 
ing new physics, and has prompted both theorists and experimentalists alike to search for 
anomalous couplings involving the top quark. On the experimental side, the CDF 0,^] and 
DO [H collaborations have begun to explore the physics of top quark rare decays, and in- 
teresting bounds on flavor- changing decays to electroweak gauge bosons have been reported 
0. On the theoretical side, a systematic examination of anomalous top quark interactions 
is being actively undertaken 

An interesting set of anomalous interactions are those given by the flavor-changing 
chromo- and electro-magnetic operators 

^g.ca^'^^tGI, + h.c. (1) 

and 

^eca^'HF^, + h.c, (2) 

where A is the new physics cutoff, tig and define the strengths of the couplings, and G^j/ 
and F^y are the gauge field tensors. The investigation of these couplings is well motivated. 
Although these operators can be induced in the standard model by high order loops, their 
effects are too small to be observable 0. However, it has been argued that they may be 
enhanced significantly in many extensions of the standard model, such as SUSY or other 
models with multiple Higgs doublets RH], models with new dynamical interactions of the 



top quark , and models where the top quark has a composite ||T0[ or soliton |Tl|] structure. 
Therefore, any observed signal indicating these types of couplings is direct evidence for 
non-standard physics and will improve our understanding of flavor dynamics. 

In this letter we propose an optimized procedure to search for the coupling teg via the 
decay t ^ eg at the Tevatron energies. We find that an improved limit on Kg, better than 



that obtained before from indirect constraints 0, is possible even based on the existing 
200 pb~^ Tevatron data in the W + 3-jet mode of t-i pair production. 

II. EXISTING LIMITS ON FLAVOR-CHANGING TOP QUARK COUPLINGS 

The strategy of searching for anomalous top quark couplings consists of two complemen- 
tary approaches: (i) obtain indirect bounds from low energy processes in which top quark 
anomalous couplings can enter via loop processes, or from the bound on t ^ bW which 
puts limits on other decay modes of the top, and (ii) direct searches at high energies for 
the effects of the anomalous couplings in top quark production and decay. We have found 
earlier [^] that the experimental lower limit on BF{t —>■ bW) from CDF 0] implies an upper 
bound of BF{t eg) < 0.45 at one standard deviation, which gives the limit \K,g\ < 0.95 



for A = 1 TeV. Then from data on 6 ^ s + 7 , we found correlated bounds on Kg and 
K^, with \k,^\ < 0.3 for \Kg\ < 0.95 and A = 1 TeV. More recently the CDF data gives 
the bound 

BF{t C7) + BF{t U7) < 2.9% (3) 



at 95% Confidence Level (CL), which translates to k^/A < {^.7?,/^BF{t bW), where A 
is in units of TeV. Reference 0] also gives the bound 

BF{t cZ) + BF{t uZ) < 90% (4) 

at 90% CL, which puts limits on the flavor-changing neutral current couplings tcZ^ and tuZ^. 



Using the anomalous coupling Ktc defined in Ref. |jT3|, which denotes the combined effect of 
V + A and V — A Ztc couphngs, we obtain a rather loose limit of < 1-3/ yjBF(t bW). 
The low energy data give ntc < 0.29 [|T^. Hence for the anomalous couplings and ntc the 
low energy data still provide tighter bounds than the direct limits from top quark decay. 
The situation for the anomalous coupling Kg is different. Because the indirect constraint 



< 0.95 is relatively weak, and since the interaction in Eq. ([^) involves the strong coupling 
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constant, there is the possibihty that it can contribute significantly to top quark production 
and decay in hadron coUiders. 



III. TOP QUARK DECAY TO CHARM-GLUON AT THE TEVATRON 

At the Fermilab Tevatron, the cross section for tt production is about 5 pb at ^/s = 
1.8 TeV. The CDF and DO experiments have each collected about 100 pb~^ of data. Since 
a significant branching fraction for t ^ eg is still allowed, the current data sample should 
be sufficient to put improved limits on Kg. Because QCD backgrounds are very large at 
hadron colliders, it is best to look for events where one top quark decays semi-leptonically 
t W^b i^ub (or i W'b i^ub) {i = e or /i) and the other decays i ^ eg (or 
t eg), i.e., 

pp ^ ti ^ i^ubcg or i~ubeg. (5) 

The signature is then £^z/) + 3 jets, where two jets (eg) reconstruct to the top mass, 

and so do the other jet (6) and the W. 

To obtain the signal event rate, we calculate the top-quark pair production via qq tt 
and gg tt using the lowest order matrix elements, and normalize the total cross section 
to theoretical results which include order corrections by including a i^'-factor of 1.4. 



For the parton distributions we use the recent parametrization MRS Set-A |]T5[. The top 
decays are calculated using exact matrix elements for each decay, assuming an on-shell W. 
We have ignored the top quark spin correlations since the top-quark production mechanisms 
we consider give insignificant top-quark polarization |TB|. The top-quark branching fractions 
may be obtained from the partial width ratio 



r(^ ^ eg) 64 



^)' = 0.91< (6) 



and thus, assuming that only t bW and t ^ eg contribute dominantly to top decays. 



where we have taken rrit = 175 GeV, as{2mt) = 0.099, A = 1 TeV, and the masses of the b 
and c quarks have been ignored. The signal cross section for the reaction given in Eq. 
can then be written as 

_ (1.82)^(k,/0.95)^ 

™""[l + 0.82(«;,/0.95)2]2' ^""^ 

where amax is the signal cross section for the maximum allowed value of the Kg (=0.95). 

Without requiring 6-tagging to begin with, we can identify the jet which goes with the 
W as follows: the pair of jets which best reconstructs the top mass is assumed to come 
from the t ^ eg decay, and the other jet is then identified as the 6-quark jet. Although the 
transverse momentum of the neutrino can be taken as the missing pj^, there is a two-fold 
ambiguity in determining the neutrino momentum along the beam direction . We choose 



the solution which best reconstructs the top mass using the momenta of the jet previously 
identified as the b jet, the charged lepton, and the neutrino. 

To make the calculation more realistic, we simulate the detector effects by assuming a 
Gaussian energy smearing for the electromagnetic and hadronic calorimetry as follows: 



AE/E = 30%/VE®l%, for leptons 

'v^©5%, for jets, (9) 



where the ® indicates that the E'-dependent and -E-independent errors are to be added in 
quadrature, and E is to be measured in GeV. 



The dominant background is from W production plus three QCD jets [0, 

pp W^jjj i^iyjjj. (10) 

Although the production rate of the background process is significantly larger than that 
of ti production at Tevatron energies, the kinematics for those processes is quite different, 
especially after imposing the top-quark mass constraint. 

To simulate the detector coverage and help reduce the background, we first impose the 
following "basic" acceptance cuts on the transverse momentum {pj^), pseudo-rapidity [rj), 



and the separation in the azimuthal angle-rapidity plane (Ai?) of the charged lepton, jets 
and missing transverse momentum 

> 15 GeV, p^>20GeV, |r7^|, |77^| < 2.5, ARej > OA, ARjj > 0.8. (11) 

The higher transverse momentum and AR cuts for the jets is motivated by the hard nature 
of the heavy top decay. The signal cross section for the maximal allowed value of Kg is 
reduced from about 550 fb with no cuts to around 322 fb with these basic cuts, while the 
1^ + 3 jets background is about 10.7 pb after the cuts. 

To improve the relative strength of the signal, we make use of the following facts: 

• the top-antitop invariant mass M{ti) has a kinematical lower limit {2mt before energy 
smearing is applied), while the lower limit is significantly smaller for the background, 
near the Wjjj threshold; 

• the final state jets in the signal have transverse momenta typically the order of ~ 
80 GeV due to the nature of top-quark two-body decay, while all the jets in the 
background events tend to be soft. We can define two scalar sums of the transverse 
momenta: 

Prijih) = 1:4' I + Wt I > Prinj) = PrUih) + (12) 

where the jet transverse momenta are ordered such that |py | > > IPtI- f^-ct, 
the signal spectra are much harder than the background at the low end, but they are 
limited by the physical scale 2mt, so that the background tends to extend relatively 
further at the high end. 

With these points in mind, we therefore accept events with 
M{ti) > 2mt = 350 GeV, 100 < Py^iis) < 300 GeV, 150 < PyOij) < 400 GeV. 

(13) 
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If we define our Level-II cuts as those given in Eq. (|T^), then the maximal signal is reduced 
only moderately to about 275 fb, while the background is reduced by about a factor of six 
to around 1870 fb. 

In Fig. 1 we show the reconstructed top-quark mass distributions M{cg) and M(bW) 
after making both the basic and Level-II cuts, where the W momentum is obtained from 
the momenta of the charged lepton and the reconstructed neutrino. We see from Fig. 1(a) 
that the continuum background is still above the M{cg) signal peak in the region around 
rrit. A further improvement can be made if we impose the cut 

\M (eg) -mt\ < 20 GeV. (14) 

In Fig. 1(b) the dashed histograms show how the background in the M{bW) distribution is 
reduced by the cut in Eq. (p!4D. The signal distribution is reduced only moderately by this 
cut (see the dotted and the solid curves) and is now within a factor of two of the background 
when Kg is at its maximal value. The signal observability can be maximized if we consider 
the events in the mass range 

\M{bW) -mt\ < 30 GeV. (15) 
After the final cut in Eq. ([T5|) , the maximal signal cross section is 

amax = 195 fb (16) 

and the background is about 400 fb. Therefore up to 40 signal events would be expected 
for the current integrated luminosity, with a background of about 80 events, which would 
correspond to nearly a 4a signal near the M{bW) peak. Table | summarizes the effect of the 
various cuts on the maximal signal and the background. The signal rate for non-maximal 
Kg is easily computed using Eqs. (|) and (|T6|). 

We have so far optimized S/B only based on kinematical variables. If we further require 



a 6-tagging on the jet that satisfies Eq. (pTsP in top-quark mass reconstruction, and assume 
a 50% 6-tagging efficiency and 1% impurity one expects to improve the S/B ratio by a 
factor of 50. 



To estimate the sensitivity to Kg for a given integrated luminosity, we can take the cross 
section given in Eq. (|) , using the value of <7max in Eq- (0) , and compare it to the background 
rate. With Gaussian statistics, a measurement is sensitive to the signal at 99% CL when 

S/^^STb = 3. (17) 

The solid line in Fig. 2 presents the anomalous coupling Hg versus the integrated luminos- 
ity required at the Tevatron with ^/s = 1.8 TeV. The dashed curve in Fig. 2 shows the 
improvement in sensitivity if 6-tagging is employed. We see that with 200 pb~^ integrated 
luminosity as accumulated by CDF and DO collaborations, one should be able to probe 
this anomalous coupling to Hg ~ 0.43 — 0.65 with or without 6-tagging, corresponding to a 
branching fraction BR{t eg) ~ 15% - 28%. In other words, if we assume the anomalous 
coupling Kg is naturally of order unity and allow the new physics cutoff scale (A) to change, 
then the current Tevatron data should be sensitive to A ~ 2 TeV. 

In the future, with ^/s = 2 TeV and the expected 1 fb^^/yr integrated luminosity of 
the Main Injector, or 10 fb^^/yr at the Tevatron Upgrade, further dramatic improvements 
in the limits on Kg should be possible. Potential results for the 2 TeV Tevatron are also 
shown in Table |I| (in parentheses) and Fig. 2 (dotted and dash-dotted curves). We see that 
a branching fraction of order 2% (0.5%) would be reached for 1 fb~^ (10 fb~^) integrated 
luminosity, corresponding to a probe of the coupling down to Kg = 0.15(0.07). 



IV. DISCUSSION AND SUMMARY 

Before concluding, a few remarks are in order. First, the top-quark events with the SM 
hadronic decay t bW bjj may pose a background to our signal as well if one of the 
jets escapes detection. However, our requirement in Eq. (|1^) for rrit reconstruction by two 
non-6 jets would hopefully remove this background. Second, if such a coupling exists at an 
observable level, there might also be a possibility of significant single top production via the 
anomalous vertex. A study of qq — * tc has recently appeared Given the fact that the 
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signal from the decay t ^ eg is more kinematically characteristic, our results here should 
be more promising. More detailed studies with all contributing processes which involve the 
teg coupling, including calculations at LHC energies, will be presented elsewhere [pT| . 

In summary, we have found that the current Tevatron data sample can already be used 
to improve the current limit on (or detect the existence of) an anomalous flavor-changing 
magnetic teg coupling via a direct search for t ^ eg decay in top-antitop pair production. 
The upgraded Tevatron will allow a probe of the t ^ eg branching fraction to the order of 
1%. 
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TABLES 

TABLE L Cross sections in units of fb for the tt — > l^ubcg signal (with maximal coupling 
Kg = 0.95), and the Wjjj background at the Tevatron with ^/s = 1.8 TeV (2 TeV). The results are 
shown at various stages of the analysis: for the basic acceptance cuts of Eq. ([TlD, after the Level-II 
cuts of Eq. ([l^ , after the cuts on M{cg) and M{bW), and finally after including 6-tagging. A 50% 
6-tagging efficiency and 1% impurity are assumed |T9t| . 



Cuts 


Signal: amax{tt i^i^bcg) (fb) 


Background: (T{W^jjj — > (.^vjjj) (fb) 


Basic Cuts 


322 (447) 


7920 (10700) 


LeveHI 


276 (380) 


1870 (2850) 


\M{cg) -mt\< 20 GeV 


239 (329) 


767 (1150) 


\M{bW)-mt\ < 30 GeV 


195 (268) 


399 (585) 


plus 6-tagging 


98 (134) 


4(6) 



FIGURE CAPTIONS 

FIG. 1 Invariant mass distributions after basic and Level-II cuts at the Tevatron with 
= 1.8 TeV for (a) M{cg) and (b) M{hW) with the additional cut \M{cg)-mt\ < 20 GeV. 
The solid curves are the signal and the dashed curves represent the VT + S jet background. In 
the M{hW) distribution, also shown are the background (upper dashed) and signal (dotted) 
curves before the M{cg) cut (the signal is about 20% lower with this cut). 

FIG. 2 Sensitivity to Kg vs. integrated luminosity at the Tevatron at 99% CL. The 
solid (dashed) curves represent the sensitivity at y/s = 1.8 TeV without (with) 6-tagging, 
and the dotted (dot-dashed) curves represent the sensitivity at i/i = 2 TeV without (with) 
6-tagging. 
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